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Abstract 



A search of the ERIC data base, Academic Search Premier, and a review of literature 
suggests that meta-analysis is ignored by career and technical education (CTE) 
researchers, a situation that is regrettable but remediable. The purpose of this paper was 
to provide CTE researchers and consumers with selected procedures and guidelines for 
conducting meta- analytic research. The highlights presented include an historical 
overview of meta-analysis, selected meta- analysis procedures, advantages of 
meta- analysis. The paper caution that CTE consumers and policy makers should be 
aware of some of the limitations associated with meta- analyses. A key assertion of the 
paper is that meta- analysis will be most valuable when it is used by those most 
knowledgeable about the substance of methodology in our field of career and technical 
education. 




Introduction 



The basis of scientific knowledge can be attributed to the accumulation of research. 
However, a single study can rarely provide a generalizable answer to research questions 
within the social sciences domain. Numerous methods have been proposed for 
cumulating results of research studies. Green and Hutchinson (1996), reported that some 
of these methods include the narrative review, counting statistically results, and various 
ways of averaging numerical outcome indices. Meta - analysis is one of the latter 
methods that is used to quantitatively synthesize research findings. 

Grover (1993) stated that: 

Meta-analysis is a method by which one attempts to quantitatively integrate 
findings from several empirical research studies related to some way to a 
common general topic. The term was officially coined in 1976 by Gene V. Glass 
in his presidential address to the American Educational Research Association, (p.3) 
According to Boston (2002), “given the growing demand for “evidenced - based 
research” to guide educational interventions, interest in the research technique of 
meta- analysis has surged” (p.l). Glass (1976) argues that “meta - analysis refers to the 
analysis of analyses. I use it to refer to the statistical analysis of a large collection of 
analysis results from individual studies for the purpose of integrating the findings” (p.3). 

The use of meta - analytic methods in educational research has been wide spread since 
the mid - twentieth century (Stewart, 2002). In their meta - analysis study on 
“The Effects of Participating in Vocational Education,” Gonzalo and Rapes (1982), noted 
that limitations were found in the methodology of the meta- evaluation, including 
inadequate conceptual framework. However, the use of meta-analysis is not 




extensively used in the field of career and technical education, as revealed from the 
review of literature. CTE research has burgeoned over last two decades; both the number 
of new journals and the proportion of research articles in them attest to this increase. It is 
important at this moment in the field to integrate findings — to clarify what it is we know 
and what it is we need to find out. Integration is essential for both CTE researchers and 
practitioners. It helps CTE researchers to direct future efforts and practitioners to identify 
teaching and management practices. 

The purpose of this paper was to provide CTE researchers, editors, and readers with 
selected procedures and guidelines for conducting meta- analytic research. 

The term meta - analysis is often used as a synonym for research synthesis, however 
in this paper, it was used to describe the procedures that CTE researchers may use to 
statistically combine the results of studies. The following objectives were formulated to 
describe the purpose of this paper: 

1. To describe the history of Meta- analysis. 

2. To describe selected Meta- analytic procedures. 

3. To describe advantages and limitations of Meta - analysis. 

Historical Overview of Meta - Analysis 
Although Glass was the first person to coin the term ‘meta - analysis” (as cited in 
(George, 2001), he was not the first person to discuss techniques for combing results. 
Early meta- analyses involved combing results from different agricultural science 
studies (Onwegbuzie & Eeach, 2003). Early in the 20‘^ century, researchers such 
as Karl Person, W. G. Cochran, and R. A. Eisher (as cited in George, 2001), 
reported that there was a need to consolidate literature in a given field. 




Karl Person conducted what was considered the first meta- analysis in 1904 
Pearson gathered data from over 10 studies pertaining to evidence on a 
vaccine against typhoid. For each study he calculated a statistic called correlation 
coefficient (Cooper, 2007). In 1932, Ronald Fisher, noted that (as cited in Cooper, 2007) 
“It sometimes happens that although few or [no statistical tests] can be claimed 
individually as significant, yet the aggregate gives an impression that the probabilities 
are lower than would have been obtained by chance” (p.3). Fisher was responsible for 
combining the p- values that were generated from statistically independent tests of the 
of the same hypotheses (Olkin, 1990). Cooper (2007) argues that these early procedures 
for combining results of independent studies were not thoroughly utilized. 

Cooper (1989) reported that meta- analysis became popularized in the 1970s and 
1980s. According to Onwuegbuzie and Leach (2003), “the technique of meta- analysis 
arose in the 1970s from concerns about imprecise characterization of findings from 
literature reviews” (p.5). The 1980s produced the appearance of several books primarily 
focusing on meta- analytic methods. The “Handbook of Research Synthesis” was 
published in 1994(Cooper & Hedges, 1994). The emphasis of this book was on rigorous 
research synthesis in the social and behavioral sciences. 

Description of Selected Meta- Analytic Procedures 
There are several elements of meta- analysis that should be addressed when 
conducting meta- analysis research. However, the purview of this section will focus 
on the following elements of meta- analysis - definition of the problem, identification of 
studies, effect size estimation, and homogeneity testing. 

Define the problem: define the variables of interest both conceptually and 




operationally. This should be done so that relevant and irrelevant studies can be 
distinguished. 

Collect the research evidence: Identify the total number of studies that have 
examined the relationship of interest. This can be done through databases such as ERIC 
and Academic Search Premier. A valid meta- analysis requires a representative survey of 
the research literature. A major problem that could occur when doing a literature search 
relates to the “file drawer problem” (Rosenthal, 1979). George (2001) reported that 
a common example of the “file drawer problem” include studies that are lost to the 
researcher in the process of doing a meta- analysis. L’Hommedieu, Menges, and Brinko 
(1988), noted that the studies that fall into the file drawer are the ones that do not show 
statistically significant differences among groups and subsequently are not accepted by 
peer reviewed journals. Other examples of “file drawer problem” are: unpublished 
research reports, research from other fields, dissertations, and master’s theses. 

Ejfect Size Reporting: After studies have been obtained and read, measures of 
effect size estimates of each finding should be calculated. The typical way of reporting 
meta- analytic results is by an index of effect size. Cohen (1969) described effect sizes 
as the “degree to which a certain phenomenon exists in a population” (p.9). Snyder and 
Lawson (1993) reported that, “effect sizes or magnitude of effects inform readers how 
much of the criterion variable can be controlled, predicted, or explained” (p.335). 
Researchers are encouraged to report their effect sizes (American Psychological 
Association, 2001; Gordon, 2001; Kotrlik & Williams, 2003, Warmbrod, 2001). 
According to Thompson (as cited in Moore, 1999), doing so, is more likely, to increase 
the chances that the research will be included in future meta- analytic studies. 




Determination of an ideal or appropriate measure of effect size is based on the data, 
and statistical models used in studies selected in the meta - analysis (George, 2001). The 
major families of effect sizes commonly used in meta- analyses are r and d. The r, is 
symbolic of the Pearson product moment correlation; measures the relationship between 
between two variables. The d, was developed by Cohen (1988), hence the 
term, Cohen’s d; measures the standardized difference between two means, and the unit 
for the size of difference is expressed in standard deviations (Quintana & Minami, 2006). 
Table 1 shows selected examples of effect size choices. 

Insert Table 1 about here 

Effect sizes are generally interpreted in the same way that z scores are interpreted. 
Cohen (1988) proposed the following descriptions: .20 is small; .50 is medium; and .80 
is large. Quintana and Minami (2006), reported that the following metrics are 
recommended for continuous variables: r, d, eta, and omega. They also suggested the 
use of odds ratio for dichotomous variables. Hommedieu, et al (1988) noted that: 
Computing effect sizes from F- and T - values will usually yield a more 
conservative individual figure, however, due to reporting and publication 
biases that favor tests showing statistically differences, the overall mean 
effect size may be inflated when T- values are used. (p.l28) 

Homogeneity Testing: Moore (1999) argues that if commonalities exist among studies, 
it is appropriate to unite estimates of effect sizes. Moore (1999), noted that “a common 
problem arises when studies do not have a common population effect size. Combining 
estimates of effect sizes for these studies would result in deception” (p.l3). Rosenthal 
and Rubin’s study (as cited in Moore, 1999), provide a test to determine homogeneity of 




effect size. Onwuegbuzie and Leech (2003) proposed using the Q-statistic as a common 
statistical technique of testing for homogeneity of effect sizes. 

Advantages of Meta- Analysis 

Sindelar and Wilson (1984, p.84) argued that meta - analysis appears to be superior 
when compared to traditional methods of summarizing literature (example the voting 
method) in several ways, such as: increased objectivity; freedom from arbitrary levels 
of statistical significance, and potential to relate independent variables to effect size. 
Wiersma (2000) noted that, “studies on the same research topic typically vary in 
measures, designs, and statistics used and meta- analysis provides a common base for 
comparing results” (p.372). 

In the field of education, meta- analyses can be easily identified because of its ERIC 
descriptor. Meta- analyses also provide extensive reference lists that assist with the 
synthesis of results. The process of meta- analysis generally generates ample information 
pertaining characteristics of the study (Wiersma, 200). According to Gay, Mills, and 
Airasian (2006): 

A central characteristic that distinguishes meta- analysis from traditional 
approaches are the emphasis placed on making the review as inclusive as possible. 
Thus, reviewers are encouraged to include results typically excluded, such as 
those presented in dissertation reports and unpublished works, (p.55) 

Kavale’s research on meta- analysis (as cited in Onwuegbuzie & Leech, 2003), revealed 
the following advantages of meta- analysis: 

1 . It uses quantitative - statistical methods for organizing and extracting 



information from large databases. 




2. It eliminates study selection bias -no pre- judgments about research quality are 
made. 

3. It makes use of all information — study findings are transformed to 
commensurable expressions of effect magnitude. 

4. It detects interactions — study characteristics that may mediate findings 
are defined measured, and their covariation is studied. 

5. It seeks general conclusions — practical simplicity that does not obscure 
an important interactive finding is sought, (pp. 11-12) 

Limitations of Meta- Analysis 

Although there are many benefits of meta- analysis, there are also some 
concerns and criticism to this technique. Several researchers (Chambers, 2004; 
George, 2001; McNeil & Newman, 1994), have documented disadvantages 
in conducting meta - analysis. One of the most common disadvantages is 
called the “apples and oranges” comparison. George (2001) suggests that this 
limitation involves the concept that, “meta- analysis is combing and comparing 
different independent variables and different dependent variables” (p.9). Some 
researchers (as cited in Chambers, 2004) suggest that, “aggregating results that 
use different research technique is inappropriate because they are too 
dissimilar” (p.36). McNeil and Newman (1994) argued that results should not 
be combined if there are significant differences. Chambers (2004) noted that it 
is fairly easy to rectify this limitation by the mechanism of coding for different 
techniques and conducting various tests to determine if outcomes are too 



dissimilar. 




A second limitation is the use of data from poorly executed studies. Validity 
issues can be a major concern with poorly designed studies. George (2000) 
stated that, “one study might use measures that yield reliable and valid 
dependent variable scores, while another might select variable measures with 
scores that is unreliable or valid” (pp.9-10). 

A third limitation is publication bias as to which articles get published. As 
noted earlier, Rosenthal (1979) described this limitation as the “file drawer 
problem.” Onwuegbuzie and Leech (2003) noted that with this limitation, 
“statistically non- significant findings are less likely to be retrieved, and thus 
less likely to be included in meta- analyses than are statistically significant 
findings” (p.l3). 

Another limitation is the use of multiple non- independent findings from the 
same study. Chambers (2004) recommends that, “it is important that one 
calculate effect sizes that are independent, so it is a common practice for 
researchers to use only one effect size for each study” (p.36). Wolf (1986) 
argued that separate meta- analyses should be conducted for different types 
of outcome variables. 

Summary and Concluding Remarks 

Meta - analysis combines data from many studies into a single index. It 
enables researchers to summarize results of many studies on a particular 
question or problem. A central characteristic that distinguishes meta- analysis 
from more traditional approaches is the emphasis placed on making the review 
as inclusive as possible. “Meta- analysis is not done by including five or so 




studies; some include upwards of sixty to seventy studies” (Wiersma, 2000, 
p.374). The most widely used index is the average effect size. Effect size is a 
numerical way of expressing the strength or magnitude of a reported 
relationship. By using the effect sizes from numerous studies that share a 
hypothesis, a researcher can make a reasonable generalization in regards to 
relationship between variables. GTE researchers are strongly urged to report 
effect sizes in manuscript and tables when reporting statistical significance. 

It is important that GTE consumers and policy makers know about some of 
methodological problems associated with meta-analyses. Meta- analysis will be 
most valuable when it is used by those knowledgeable about the substance of 
methodology of research in our field. 

Kavale (2001) stated that: 

Euture research is quiet likely to remain a generally unorganized, 
decentralized, and non- standardized process without regard to how it may 
fit together into a comprehensive whole. Eor this reason, we will likely 
continue to need research synthesis methods, such as meta- analysis, 
to make research findings believable. (p.l82) 

Although the theory of meta- analysis is complex, application of the method is 
relatively simple, particularly considering the benefits that can be derived from 
it. Researchers in GTE have generated much data over the years. Therefore, it 
is now time to apply meta - analysis to draw conclusions that will stand the test 



of time. 
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Table 1 

Selected Examples of Effect Size Choices 



Effect Size 


Common Application 


Formula 




regression 


S S explained/s S total 


Omega^(tjO^) 


ANOVA 


(5'5'bETWEEN - - 1) X MSwitHIn)/(SStoTAL + -^5'within)> 

Where k is the number of groups 


Cohen’s d 


toxANOVA 


(-^EXPERIMENTAL ~ -^CONTROl)/5'C)poolED 




